Multi-objective Feature Selection Method Research Articles

A Multi-objective Feature Selection Method Considering the Interaction Between Features

A Multi-objective Feature Selection Method Considering the Interaction Between Features

A Multiform Optimization Framework for Multi-objective Feature Selection in Classification

Feature selection in machine learning as a key data processing technique has two conflicting goals: minimizing the classification error rate and minimizing the number of features selected. However, most of the existing multi-objective feature selection methods face the problems of easily falling into local optima and slow convergence by virtue of their problem characteristics, such as partially conflicting objectives and highly discontinuous Pareto fronts. To solve these problems, this article proposes a multiform optimization framework to solve a multi-objective feature selection task together with several auxiliary single-objective feature selection tasks in a multitask environment. The proposed framework uses the problem-solving experience of single-objective tasks to assist the multi-objective feature selection task in exploring more promising regions and accelerating the convergence speed. Specifically, a knowledge transfer strategy based on the search experience of different tasks is developed to accomplish multiform optimization. In addition, a diversity enhancement mechanism is presented to improve search ability in promising decision space areas by considering historical information about the population. In most cases, the experiment results on 27 datasets demonstrate that the proposed technique can uncover more diversified feature subsets on the Pareto front in less time than existing state-of-the-art methods.

A novel enterprise credit risk rating model with the perspective of multiple regulatory demands

With the number of market participants rising, the issue of contradiction between heavy supervision task and insufficient supervision resource have become increasingly urgent for regulatory authorities to consider. The credit risk rating of enterprises is therefore introduced to improve the efficiency of supervision. By analyzing the purposes of enterprise credit risk rating and the multiple demands for regulatory authorities, this paper offers a multi-classification ensemble model with hybrid decomposition strategy and two-stage multi-objective feature selection. First, an adaptive resampling technique is introduced into the hybrid decomposition strategy for multi-classification, so to appropriately allocate supervision resources in each credit risk class. Then, a two-stage multi-objective feature selection method is developed to combine the various demands of regulatory authorities for feature selection. In the first stage, results of multiple Filter feature selection methods are integrated as output to improve the performance of feature information; In the second stage, the feature information from the first stage is used as input to guide the Wrapper feature selection, which is based on the Improved Multi-objective Ant Colony Optimization. Finally, a real listed enterprise credit dataset of China is used to evaluate the performance of each module and the entire model. The empirical results show that the design of hybrid decomposition strategy and multi-objective feature selection can not only deal with the class-imbalanced problem during the decomposition process, but also take into account the balance of each demand well. More importantly, a desirable framework is set in this paper to simplify the process from imbalanced multi-classification to a balanced binary one.

A Novel Multi-Objective Based Feature Selection Method for Response Prediction

Accurate response prediction is essential towards personalized treatment in radiation therapy. Excessive imaging features, extracted from medical images, pose a great challenge in radiomic analyses. Feature selection is an essential step to remove redundant and irrelevant features for model construction. We proposed a novel multi-objective based radiomic feature selection method (MRMOPSO), where the number of features, sensitivity, and specificity are jointly considered as optimization objectives for feature selection. The MRMOPSO innovated by three aspects: 1) Fisher score initialize the feature population to speed up the convergence; 2) Min-redundancy particle generation operations to reduce the redundancy between radiomic features, a truncation strategy was also introduced; 3) Particle selection operation guided by elitism strategies to improve local search ability of the algorithm. We evaluated the effectiveness of the proposed MRMOPSO method using a cohort of oropharyngeal cancer patients from The Cancer Imaging Archive (TCIA). 357 patients were used for model training and additional 64 patients were used for independent evaluation. The proposed methods were compared with (a) classical feature selection methods, i.e., Lasso, minimal-redundancy-maximal-relevance criterion (mRMR), F-score, and mutual information (MI), (b) single-objective feature selection methods, i.e., genetic algorithm (GA), particle swarm optimization algorithm (PSO) and (c) multi-objective feature selection methods, i.e., multiple objective particle swarm optimization (MOPSO), nondominated sorting genetic algorithm II (NSGA II). The other feature selection methods yielded AUCs, sensitivity, specificity of (0.48-0.71), (0.49-0.86), (0.33-0.67), respectively. The MRMOPSO achieved significantly highly AUC of 0.84 with smaller number of selected features on the independent dataset (Table 1). Additionally, the MRMOPSO remarkably improved the sensitivity (0.81), specificity (0.81) and achieved an excellent balance between sensitively and specificity. We demonstrated a novel multi-objective based radiomic feature selection method. The proposed algorithm effectively reduced feature dimension, and achieved superior AUC with simultaneous improved sensitivity and specificity, for radiomic response prediction.

Differential Evolution-Based Feature Selection: A Niching-Based Multiobjective Approach

Feature selection is to reduce both the dimensionality of data and the classification error rate (i.e., increase the classification accuracy) of a learning algorithm. The two objectives are often conflicting, thus a multiobjective feature selection method can obtain a set of nondominated feature subsets. Each solution in the set has a different size and a corresponding classification error rate. However, most existing feature selection algorithms have ignored that, for a given size, there can be different feature subsets with very similar or the same accuracy. This article introduces a niching-based multiobjective feature selection method that simultaneously minimizes the number of selected features and the classification error rate. The proposed method conceives to identify: 1) a set of feature subsets with good convergence and distribution and 2) multiple feature subsets choosing the same number of features with almost the same lowest classification error rate. The contributions of this article are threefold. First, a niching and global interaction mutation operator is proposed that can produce promising feature subsets. Second, a newly developed environmental selection mechanism allows equal informative feature subsets to be stored by relaxing the Pareto-dominance relationship. Finally, the proposed subset repairing mechanism can generate better feature subsets and further remove the redundant features. The proposed method is compared against seven multiobjective feature selection algorithms on 19 datasets, including both binary and multiclass classification tasks. The results show that the proposed method can evolve a rich and diverse set of nondominated solutions for different feature selection tasks, and their availability helps in understanding the relationships between features.

A multi-objective based radiomics feature selection method for response prediction following radiotherapy

Objective. Radiomics contains a large amount of mineable information extracted from medical images, which has important significance in treatment response prediction for personalized treatment. Radiomics analyses generally involve high dimensions and redundant features, feature selection is essential for construction of prediction models. Approach. We proposed a novel multi-objective based radiomics feature selection method (MRMOPSO), where the number of features, sensitivity, and specificity are jointly considered as optimization objectives in feature selection. The MRMOPSO innovated in the following three aspects: (1) Fisher score to initialize the population to speed up the convergence; (2) Min-redundancy particle generation operations to reduce the redundancy between radiomics features, a truncation strategy was introduced to further reduce the number of features effectively; (3) Particle selection operations guided by elitism strategies to improve local search ability of the algorithm. We evaluated the effectiveness of the MRMOPSO by using a multi-institution oropharyngeal cancer dataset from The Cancer Imaging Archive. 357 patients were used for model training and cross validation, an additional 64 patients were used for evaluation. Main results. The area under the curve (AUC) of our method achieved AUCs of 0.82 and 0.84 for cross validation and independent dataset, respectively. Compared with classical feature selection methods, the AUC of MRMOPSO is significantly higher than the Lasso (AUC = 0.74, p-value = 0.02), minimal-redundancy-maximal-relevance criterion (mRMR) (AUC = 0.73, p-value = 0.05), F-score (AUC = 0.48, p-value < 0.01), and mutual information (AUC = 0.69, p-value < 0.01) methods. Compared to single-objective methods, the AUC of MRMOPSO is 12% higher than those of the genetic algorithm (GA) (AUC = 0.68, p-value = 0.02) and particle swarm optimization algorithm (AUC = 0.72, p-value = 0.05) methods. Compared to other multi-objective feature selection methods, the AUC of MRMOPSO is 14% higher than those of multiple objective particle swarm optimization (MOPSO) (AUC = 0.68, p-value = 0.02) and nondominated sorting genetic algorithm II (NSGA2) (AUC = 0.70, p-value = 0.03). Significance. We proposed a multi-objective based radiomics feature selection method. Compared to conventional feature reduction algorithms, the proposed algorithm effectively reduced feature dimension, and achieved superior performance, with improved sensitivity and specificity, for response prediction in radiotherapy.

Feature Selection Using Diversity-Based Multi-objective Binary Differential Evolution

By identifying relevant features from the original data, feature selection methods can maintain or improve the classification accuracy and reduce the dimensionality. Recently, many multi-objective evolutionary methods have been proposed for feature selection. However, effectively handling the trade-offs between convergence and diversity of the non-dominated solutions remains a major challenge, especially for high-dimensional datasets. To cover this issue, this work studies a diversity-based multi-objective differential evolution approach to feature selection. During the environmental selection process, each of the solutions in the candidate pool will have a diversity score, and solutions with large diversity score values will be preferred so as to improve the population diversity. To reduce the search space, irrelevant and weakly relevant features are detected and removed in the proposed method. A new binary mutation operator using the neighborhood information of individuals is also proposed, aiming to produce better feature subsets. Experimental results on 14 datasets with varying difficulties show that the proposed feature selection method can obtain significantly better feature selection performance than current popular multi-objective feature selection methods.

Applications of dynamic feature selection and clustering methods to medical diagnosis

Machine learning methods are commonly used for disease and cancer diagnosis. The model performance can be improved via feature selection, feature reduction, and clustering methods. Although these supplementary techniques have certain advantages, they cannot necessarily guarantee better performance. The objective of this study is to improve the performance of classification methods used for medical diagnosis of various diseases. We propose a dynamic feature selection method based on the merits of both principal component analysis and Wrapper feature selection methods. It is a novel multi-objective feature selection method based on a customized genetic algorithm that is guided by eigenvalues of the features and feedbacks of various classifiers’ output. To reinforce classification learning and further enhancement of their performance, we utilize a dynamic selection of three clustering methods including K-means, fuzzy c-means, and particle swarm optimization. We also investigated the performance of two deep learning classifiers on the proposed methods.To show the impacts of combination of the proposed methods, we analyze the results of applying 12 machine learning and two deep learning classifiers to 30 imbalanced medical datasets. According to our extensive computational experiments and the statistical tests, the proposed dynamic feature selection and clustering methods perform significantly better than existing methods. The proposed methods not only improve the average of performance measures by 5% but also are more accurate than best performing classification methods in the literature used for the same datasets.

Multi-Objective Feature Selection Method by Using ACO with PSO Algorithm for Breast Cancer Detection

Multi-Objective Feature Selection Method by Using ACO with PSO Algorithm for Breast Cancer Detection

A multi-objective feature selection method using Newton’s law based PSO with GWO

In high dimensional data, the Feature Selection (FS) approach plays an important role in overcoming accuracy, time complexity, and space complexity. This paper proposes a binary version of the hybrid two-phase multi-objective FS approach, based on Particle Swarm Optimization (PSO) and Gray Wolf Optimization (GWO). The first objective is to minimize the classification error rate, and the second objective is to reduce the number of selected features. In the first phase, the proposed approach performs the global search, and in the second phase, it goes for the local search. In the case of global search, i.e., for the exploration phase, the property of PSO is used. In the case of local search, i.e., for the exploitation phase, the proposed method uses the modified version of PSO and GWO. The particles are learning not only from the positions but also from the mass and acceleration, based upon Newton’s second law of motion. In some cases, while in the iterations for updation of particles, a stall arises. Due to which the particles are not updating their positions. To avoid the stall in the iterations, this paper introduces a new term, i.e., population factor, which can omit the stall in the iterations if it arises. The prominent features which are selected from the proposed approach are tested on five well-known classification algorithms. The performance of the proposed approach is investigated on eight high-dimensional gene expression data. Experimental results comparison shows that the proposed approach for FS performs efficiently and effectively than other metaheuristics, statistical, and multi-objective FS methods.

A novel binary many-objective feature selection algorithm for multi-label data classification

For Multi-label classification, redundant and irrelevant features degrade the performance of classification. To select the best features based on several conflicting objectives, feature selection can be modeled as a large-scale optimization problem. However, most existing multi-objective feature selection methods select the features based on minimizing two well-known objectives, the number of features and classification error, additional objectives can be considered to improve the classification performance. In this study, for the first time, a many-objective optimization method is proposed to select the efficient features for multi-label classification based on not only two mentioned objectives, but also maximizing the correlation between features and labels and minimizing the computational complexity of features. Maximizing the correlation could lead to increasing the accuracy of classification. On the other hand, selecting less complex features decreases the computational complexity of feature extraction phase. The most important aim of this paper is to tackle the multi-label feature selection based on the number of features, classification error, correlation between features and labels, and computational complexity of features, simultaneously. The conducted many-objective feature selection problem is solved using a proposed binary version of NSGA-III algorithm. The binary operator improves the exploration power of optimizer to search the large-scale space. In order to evaluate the proposed algorithm (called binary NSGA-III), a benchmarking experiments is conducted on eight multi-label datasets in terms of several multi-objective assessment metrics, including Hypervolume indicator, Pure Diversity, and Set-coverage. Experimental results show significant improvements for proposed method in comparison with other algorithms.

Incorporation of multimodal multiobjective optimization in designing a filter based feature selection technique

Current work reports about the development of a new feature selection technique fusing two concepts, multimodal multiobjective optimization and filter-based feature selection. Use of the concept of multimodality in multiobjective based feature selection helps in generating diverse set of feature subsets on final Pareto front. This approach evaluates the quality of the reduced feature set by utilizing different quality measures. This process of feature selection focuses on achieving two discrete objectives: (1) identification of a large number of Pareto-optimal solutions along with achieving a good distribution in both objective and decision spaces; (2) making a selection of feature subset with minimal redundancy and high correlation with classes. To achieve the second objective, a variety of objective functions based on information-theoretic measures like normalized mutual information and correlation with class attributes are utilized. A multiobjective ring-based Particle swarm Optimization (PSO) and non-dominated sorting with special crowding distance are employed to cover the aspects corresponding to the first objective. An evaluation is carried out on seven publicly available datasets concerning different classifiers. The results of these experiments illustrate that the multimodal PSO based feature selection approach finds more feature subsets than its simple PSO counterpart in multiobjective environment. And, the results are also compared with those of existing wrapper based multimodal multiobjective feature selection methods.

Feature selection schema based on game theory and biology migration algorithm for regression problems

Many real-world datasets nowadays are of regression type, while only a few dimensionality reduction methods have been developed for regression problems. On the other hand, most existing regression methods are based on the computation of the covariance matrix, rendering them inefficient in the reduction process. Therefore, a BMA-based multi-objective feature selection method, GBMA, is introduced by incorporating the Nash equilibrium approach. GBMA is intended to maximize model accuracy and minimize the number of features through a less complex procedure. The proposed method is composed of four steps. The first step involves defining three players, each of which is trying to improve its objective function (i.e., model error, number of features, and precision adjustment). The second step includes clustering features based on the correlation therebetween and detecting the most appropriate ordering of features to enhance cluster efficiency. The third step comprises extracting a new feature from each cluster based on various weighting methods (i.e., moderate, strict, and hybrid). Finally, the fourth step encompasses updating players based on stochastic search operators. The proposed GBMA strategy explores the search space and finds optimal solutions in an acceptable amount of time without examining every possible solution. The experimental results and statistical tests based on ten well-known datasets from the UCI repository proved the high performance of GBMA in selecting features for solving regression problems.

Integration of multi-objective PSO based feature selection and node centrality for medical datasets

In the past decades, the rapid growth of computer and database technologies has led to the rapid growth of large-scale medical datasets. On the other, medical applications with high dimensional datasets that require high speed and accuracy are rapidly increasing. One of the dimensionality reduction approaches is feature selection that can increase the accuracy of the disease diagnosis and reduce its computational complexity. In this paper, a novel PSO-based multi objective feature selection method is proposed. The proposed method consists of three main phases. In the first phase, the original features are showed as a graph representation model. In the next phase, feature centralities for all nodes in the graph are calculated, and finally, in the third phase, an improved PSO-based search process is utilized to final feature selection. The results on five medical datasets indicate that the proposed method improves previous related methods in terms of efficiency and effectiveness.

Multi-Objective Feature Selection Method Based on Hybrid MI and PSO Algorithm

Multi-Objective Feature Selection Method Based on Hybrid MI and PSO Algorithm

A multi-objective feature selection method based on bacterial foraging optimization

Feature selection plays an important role in data preprocessing. The aim of feature selection is to recognize and remove redundant or irrelevant features. The key issue is to use as few features as possible to achieve the lowest classification error rate. This paper formulates feature selection as a multi-objective problem. In order to address feature selection problem, this paper uses the multi-objective bacterial foraging optimization algorithm to select the feature subsets and k-nearest neighbor algorithm as the evaluation algorithm. The wheel roulette mechanism is further introduced to remove duplicated features. Four information exchange mechanisms are integrated into the bacteria-inspired algorithm to avoid the individuals getting trapped into the local optima so as to achieve better results in solving high-dimensional feature selection problem. On six small datasets and ten high-dimensional datasets, comparative experiments with different conventional wrapper methods and several evolutionary algorithms demonstrate the superiority of the proposed bacteria-inspired based feature selection method.

QSAR Modelling to Identify LRRK2 Inhibitors for Parkinson's Disease.

Parkinson’s disease is one of the most common neurodegenerative illnesses in older persons and the leucine-rich repeat kinase 2 (LRRK2) is an auspicious target for its pharmacological treatment. In this work, quantitative structure–activity relationship (QSAR) models for identification of putative inhibitors of LRRK2 protein are developed by using an in-house chemical library and several machine learning techniques. The methodology applied in this paper has two steps: first, alternative subsets of molecular descriptors useful for characterizing LRRK2 inhibitors are chosen by a multi-objective feature selection method; secondly, QSAR models are learned by using these subsets and three different strategies for supervised learning. The qualities of all these QSAR models are compared by classical metrics and the best models are discussed in statistical and physicochemical terms.

Toward a reliable anomaly-based intrusion detection in real-world environments

A popular approach for detecting network intrusion attempts is to monitor the network traffic for anomalies. Extensive research effort has been invested in anomaly-based network intrusion detection using machine learning techniques; however, in general these techniques remain a research topic, rarely being used in real-world environments. In general, the approaches proposed in the literature lack representative datasets and reliable evaluation methods that consider real-world network properties during the system evaluation. In general, the approaches adopt a set of assumptions about the training data, as well as about the validation methods, rendering the created system unreliable for open-world usage. This paper presents a new method for creating intrusion databases. The objective is that the databases should be easy to update and reproduce with real and valid traffic, representative, and publicly available. Using our proposed method, we propose a new evaluation scheme specific to the machine learning intrusion detection field. Sixteen intrusion databases were created, and each of the assumptions frequently adopted in studies in the intrusion detection literature regarding network traffic behavior was validated. To make machine learning detection schemes feasible, we propose a new multi-objective feature selection method that considers real-world network properties. The results show that most of the assumptions frequently applied in studies in the literature do not hold when using a machine learning detection scheme for network-based intrusion detection. However, the proposed multi-objective feature selection method allows the system accuracy to be improved by considering real-world network properties during the model creation process.

MULTI-OBJECTIVE EVOLUTIONARY ALGORITHMS FOR FILTER BASED FEATURE SELECTION IN CLASSIFICATION

Feature selection is a multi-objective problem with the two main conflicting objectives of minimising the number of features and maximising the classification performance. However, most existing feature selection algorithms are single objective and do not appropriately reflect the actual need. There are a small number of multi-objective feature selection algorithms, which are wrapper based and accordingly are computationally expensive and less general than filter algorithms. Evolutionary computation techniques are particularly suitable for multi-objective optimisation because they use a population of candidate solutions and are able to find multiple non-dominated solutions in a single run. However, the two well-known evolutionary multi-objective algorithms, non-dominated sorting based multi-objective genetic algorithm II (NSGAII) and strength Pareto evolutionary algorithm 2 (SPEA2) have not been applied to filter based feature selection. In this work, based on NSGAII and SPEA2, we develop two multi-objective, filter based feature selection frameworks. Four multi-objective feature selection methods are then developed by applying mutual information and entropy as two different filter evaluation criteria in each of the two proposed frameworks. The proposed multi-objective algorithms are examined and compared with a single objective method and three traditional methods (two filters and one wrapper) on eight benchmark datasets. A decision tree is employed to test the classification performance. Experimental results show that the proposed multi-objective algorithms can automatically evolve a set of non-dominated solutions that include a smaller number of features and achieve better classification performance than using all features. NSGAII and SPEA2 outperform the single objective algorithm, the two traditional filter algorithms and even the traditional wrapper algorithm in terms of both the number of features and the classification performance in most cases. NSGAII achieves similar performance to SPEA2 for the datasets that consist of a small number of features and slightly better results when the number of features is large. This work represents the first study on NSGAII and SPEA2 for filter feature selection in classification problems with both providing field leading classification performance.

A multi-objective particle swarm optimisation for filter-based feature selection in classification problems

Feature selection has the two main objectives of minimising the classification error rate and the number of features. Based on binary particle swarm optimisation (BPSO), we develop two novel multi-objective feature selection frameworks for classification, which are multi-objective binary PSO using the idea of non-dominated sorting (NSBPSO) and multi-objective binary PSO using the ideas of crowding, mutation and dominance (CMDBPSO). Four multi-objective feature selection methods are then developed by applying mutual information and entropy as two different filter evaluation criteria in each of the proposed frameworks. The proposed algorithms are examined and compared with a single objective method on eight benchmark data sets. Experimental results show that the proposed multi-objective algorithms can evolve a set of solutions that use a smaller number of features and achieve better classification performance than using all features. In most cases, NSBPSO achieves better results than the single objective algorithm and CMDBPSO outperforms all other methods mentioned above. This work represents the first study on multi-objective BPSO for filter-based feature selection.